Manifold Optimization-Assisted Gaussian Variational Approximation

Gaussian variational approximation is a popular methodology to approximate posterior distributions in Bayesian inference, especially high-dimensional and large data settings. To control the computational cost, while being able capture correlations among variables, low rank plus diagonal structure was introduced previous literature for covariance matrix. For specific learning task, uniqueness of solution usually ensured by imposing stringent constraints on parameterized matrix, which could break down during optimization process. In this article, we consider two special structures applying Stiefel manifold Grassmann constraints, address difficulty such factorization architectures. speed up updating process with minimum hyperparameter-tuning efforts, design new schemes Riemannian stochastic gradient descent methods compare them other existing optimizing manifolds. addition fixing identification issue, results from both simulation empirical experiments prove ability proposed obtaining competitive accuracy comparable converge large-scale tasks. Supplementary materials article are available online.